Kanji-to-Hiragana conversion based on a length-constrained n-gram analysis

Length-Constrained N-Gram Analysis by, Joe Picone, Tom Staples, Kazz Kondo, and Nozomi Arai Tsukuba Research and Development Center Texas Instruments Tsukuba, Japan Abstract A common problem in speech processing is the conversion of the written form of a language to a set of phonetic symbols representing the pronunciation. In this paper we focus on an aspect of this problem specific to the Japanese language. Written Japanese consists of a mixture of three types of symbols: kanji, hiragana and katakana. We describe an algorithm for converting conventional Japanese orthography to a hiragana-like symbol set that closely approximates the most common pronunciation of the text. The algorithm is based on two hypotheses: (1) the correct reading of a kanji character can be determined by examining a small number of adjacent characters; (2) the number of such combinations required in a dictionary is manageable. The algorithm described here converts the input text by selecting the most probable sequence of orthographic units (n-grams) that can be concatenated to form the input text. In closed-set testing, the n-gram algorithm was shown to provide better performance than several public domain algorithms, achieving a sentence error rate of 3% on a wide range of text material. Though the focus of this paper is written Japanese, the pattern matching algorithm described here has applications to similar problems in other languages. EDICS: (1) SA 1.3.1; (2) SA 1.9 Please direct all correspondence to: Dr. Joseph Picone, Institute for Signal and Information Processing, Mississippi State University, Box 9571, Mississippi State, MS 39762, Tel: (601) 325-3149, Fax: (601) 325-2298, email: [email protected].